Talking face generation aims at generating photo-realistic video portraits of a target person driven by input audio. Due to its nature of one-to-many mapping from the input audio to the output video (e.g., one speech content may have multiple feasible visual appearances), learning a deterministic mapping like previous works brings ambiguity during training, and thus causes inferior visual results. Although this one-to-many mapping could be alleviated in part by a two-stage framework (i.e., an audio-to-expression model followed by a neural-rendering model), it is still insufficient since the prediction is produced without enough information (e.g., emotions, wrinkles, etc.). In this paper, we propose MemFace to complement the missing information with an implicit memory and an explicit memory that follow the sense of the two stages respectively. More specifically, the implicit memory is employed in the audio-to-expression model to capture high-level semantics in the audio-expression shared space, while the explicit memory is employed in the neural-rendering model to help synthesize pixel-level details. Our experimental results show that our proposed MemFace surpasses all the state-of-the-art results across multiple scenarios consistently and significantly.

虽然先前以语音为导向的说话面部生成方法在改善合成视频的视觉质量和唇部同步质量方面取得了重大进展，但它们对唇部运动的关注较少，从而极大地破坏了说话面部视频的真实性。是什么导致运动烦恼，以及如何减轻问题？在本文中，我们基于最先进的管道对运动抖动问题进行系统分析，该管道使用3D面表示桥接输入音频和输出视频，并通过一系列有效的设计来改善运动稳定性。我们发现，几个问题可能会导致综合说话的面部视频中的烦恼：1）输入3D脸部表示的烦恼； 2）训练推导不匹配； 3）视频帧之间缺乏依赖建模。因此，我们提出了三种有效的解决方案来解决此问题：1）我们提出了一个基于高斯的自适应平滑模块，以使3D面部表征平滑以消除输入中的抖动； 2）我们在训练中对神经渲染器的输入数据增加了增强的侵蚀，以模拟推理中的变形以减少不匹配； 3）我们开发了一个音频融合的变压器生成器，以模拟视频帧之间的依赖性。此外，考虑到没有现成的指标来测量说话面部视频中的运动抖动，我们设计了一个客观的度量标准（运动稳定性指数，MSI），可以通过计算方差加速度的倒数来量化运动抖动。广泛的实验结果表明，我们方法对运动稳定的面部视频生成的优越性，其质量比以前的系统更好。

在“知识图”（kgs）的表示领域中，超级关系的事实由主要三重和几个辅助属性描述组成，这被认为比基于三重的事实更全面，更具体。但是，由于代表实体之间的隶属关系的层次结构削弱，因此，单个视图中现有的超相关KG嵌入方法受到限制。为了打破这一限制，我们提出了一个双视性超相关kg（DH-kg）结构，该结构包含实体的超相关实例视图，以及对从实体到共同模型超相关的概念的超相关本体论视图和分层信息。在本文中，我们首先定义了DH-KG上的链接预测和实体键入任务，并根据医疗数据构建了两个DH-KG数据集，即从Wikidata和HTDM中提取的JW44K-6K。此外，我们根据Gran编码器，HGNN和联合学习提出了DH-KG嵌入模型DHGE。实验结果表明，DHGE在DH-KG上的表现优于基线模型。我们还提供了该技术在高血压药物领域中应用的示例。我们的模型和数据集公开可用。

由于样本量有限，可以准确估计研究地点（例如医院）中的个性化治疗效果。此外，隐私考虑和缺乏资源阻止站点利用其他站点的主题级数据。我们提出了一种基于树的模型平均方法，以通过利用从其他潜在异质部位得出的模型来提高目标部位条件平均治疗效果（CATE）的估计精度，而无需共享主题级数据。据我们的最佳知识，没有建立的模型平均分布式数据的方法，重点是改善治疗效果的估计。具体而言，在分布式数据网络下，我们的框架提供了一个基于CATE估算器的基于可解释的树的合奏，该集合可以跨研究站点加入模型，同时通过站点分区积极地对数据源中的异质性进行建模。通过对氧疗法对医院存活率的因果影响的现实研究证明了这种方法的表现，并得到了全面的模拟结果的支持。

Although many studies have successfully applied transfer learning to medical image segmentation, very few of them have investigated the selection strategy when multiple source tasks are available for transfer. In this paper, we propose a prior knowledge guided and transferability based framework to select the best source tasks among a collection of brain image segmentation tasks, to improve the transfer learning performance on the given target task. The framework consists of modality analysis, RoI (region of interest) analysis, and transferability estimation, such that the source task selection can be refined step by step. Specifically, we adapt the state-of-the-art analytical transferability estimation metrics to medical image segmentation tasks and further show that their performance can be significantly boosted by filtering candidate source tasks based on modality and RoI characteristics. Our experiments on brain matter, brain tumor, and white matter hyperintensities segmentation datasets reveal that transferring from different tasks under the same modality is often more successful than transferring from the same task under different modalities. Furthermore, within the same modality, transferring from the source task that has stronger RoI shape similarity with the target task can significantly improve the final transfer performance. And such similarity can be captured using the Structural Similarity index in the label space.

Modern deep neural networks have achieved superhuman performance in tasks from image classification to game play. Surprisingly, these various complex systems with massive amounts of parameters exhibit the same remarkable structural properties in their last-layer features and classifiers across canonical datasets. This phenomenon is known as "Neural Collapse," and it was discovered empirically by Papyan et al. \cite{Papyan20}. Recent papers have theoretically shown the global solutions to the training network problem under a simplified "unconstrained feature model" exhibiting this phenomenon. We take a step further and prove the Neural Collapse occurrence for deep linear network for the popular mean squared error (MSE) and cross entropy (CE) loss. Furthermore, we extend our research to imbalanced data for MSE loss and present the first geometric analysis for Neural Collapse under this setting.

Domain adaptive detection aims to improve the generalization of detectors on target domain. To reduce discrepancy in feature distributions between two domains, recent approaches achieve domain adaption through feature alignment in different granularities via adversarial learning. However, they neglect the relationship between multiple granularities and different features in alignment, degrading detection. Addressing this, we introduce a unified multi-granularity alignment (MGA)-based detection framework for domain-invariant feature learning. The key is to encode the dependencies across different granularities including pixel-, instance-, and category-levels simultaneously to align two domains. Specifically, based on pixel-level features, we first develop an omni-scale gated fusion (OSGF) module to aggregate discriminative representations of instances with scale-aware convolutions, leading to robust multi-scale detection. Besides, we introduce multi-granularity discriminators to identify where, either source or target domains, different granularities of samples come from. Note that, MGA not only leverages instance discriminability in different categories but also exploits category consistency between two domains for detection. Furthermore, we present an adaptive exponential moving average (AEMA) strategy that explores model assessments for model update to improve pseudo labels and alleviate local misalignment problem, boosting detection robustness. Extensive experiments on multiple domain adaption scenarios validate the superiority of MGA over other approaches on FCOS and Faster R-CNN detectors. Code will be released at https://github.com/tiankongzhang/MGA.

In this paper we derive a PAC-Bayesian-Like error bound for a class of stochastic dynamical systems with inputs, namely, for linear time-invariant stochastic state-space models (stochastic LTI systems for short). This class of systems is widely used in control engineering and econometrics, in particular, they represent a special case of recurrent neural networks. In this paper we 1) formalize the learning problem for stochastic LTI systems with inputs, 2) derive a PAC-Bayesian-Like error bound for such systems, 3) discuss various consequences of this error bound.

Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, their iterative refinement process in high-dimensional data space results in slow inference speed, which restricts their application in real-time systems. Previous works have explored speeding up by minimizing the number of inference steps but at the cost of sample quality. In this work, to improve the inference speed for DDPM-based TTS model while achieving high sample quality, we propose ResGrad, a lightweight diffusion model which learns to refine the output spectrogram of an existing TTS model (e.g., FastSpeech 2) by predicting the residual between the model output and the corresponding ground-truth speech. ResGrad has several advantages: 1) Compare with other acceleration methods for DDPM which need to synthesize speech from scratch, ResGrad reduces the complexity of task by changing the generation target from ground-truth mel-spectrogram to the residual, resulting into a more lightweight model and thus a smaller real-time factor. 2) ResGrad is employed in the inference process of the existing TTS model in a plug-and-play way, without re-training this model. We verify ResGrad on the single-speaker dataset LJSpeech and two more challenging datasets with multiple speakers (LibriTTS) and high sampling rate (VCTK). Experimental results show that in comparison with other speed-up methods of DDPMs: 1) ResGrad achieves better sample quality with the same inference speed measured by real-time factor; 2) with similar speech quality, ResGrad synthesizes speech faster than baseline methods by more than 10 times. Audio samples are available at https://resgrad1.github.io/.

Function approximation (FA) has been a critical component in solving large zero-sum games. Yet, little attention has been given towards FA in solving \textit{general-sum} extensive-form games, despite them being widely regarded as being computationally more challenging than their fully competitive or cooperative counterparts. A key challenge is that for many equilibria in general-sum games, no simple analogue to the state value function used in Markov Decision Processes and zero-sum games exists. In this paper, we propose learning the \textit{Enforceable Payoff Frontier} (EPF) -- a generalization of the state value function for general-sum games. We approximate the optimal \textit{Stackelberg extensive-form correlated equilibrium} by representing EPFs with neural networks and training them by using appropriate backup operations and loss functions. This is the first method that applies FA to the Stackelberg setting, allowing us to scale to much larger games while still enjoying performance guarantees based on FA error. Additionally, our proposed method guarantees incentive compatibility and is easy to evaluate without having to depend on self-play or approximate best-response oracles.